Page 1st pages.indd

This chapter is concerned with how the processing power and multimedia capabilities of modern computers can be exploited to understand the workings of stories told in combinations of different types of media. Since the early days of digital technology, computers have been used to count and analyze words and patterns of words in texts, chiefl y under the rubric of corpus stylistics, and more generally in the fi eld of digital humanities. At the risk of oversimplifi cation, much of this work can be characterized by its focus on single literary texts or on the works of single authors, and also by the way in which scholars look for signs of previously hypothesized linguistic or literary phenomena in the texts, which sometimes involves manual annotation of the texts prior to automated analysis. In contrast, this study advocates a novel computer-based approach to the analysis of narrative and multimodality that is characterized by the use of a computer to extract unusually frequent patterns from the surface forms of large collections of multimodal stories. Crucially, these patterns are to be extracted without the cost and bias due to prior manual annotation and the encoding of grammars, pragmatics, and world knowledge. Instead, patterns are extracted from corpora of multimodal texts on purely statistical grounds. The extraction of such patterns is envisioned as a starting point for developing theory in two ways. Firstly, by asking why particular forms are prevalent in a corpus, we may be led to develop explanations of the observed phenomena. Secondly, because the phenomena of interest have known surface forms, it is straightforward to automatically generate more data about their occurrence in the initial corpus and in new instances of multimodal texts in order to test hypotheses. By means of two case studies I assess the potential for this kind of approach, discuss its inherent limitations, and point to opportunities for further work. A major obstacle to the computer-based analysis of multimodal texts is the current limit on what can be achieved with automatic image and video analysis techniques, compared with text analysis. This issue was sidestepped in the work presented in the fi rst case study as feature fi lms were analyzed by extracting patterns